[gpt-oss][2/N] Support input_messages in responsesRequest by qandrew · Pull Request #26962 · vllm-project/vllm

qandrew · 2025-10-16T00:17:39Z

Purpose

in the RepsonsesResponse object, we already have input_messages and output_messages. We should be able to do multi_turl by allowing ResponsesRequest to have previous_input_messages as payload.

Test Plan

I tested following OpenAI's example: https://platform.openai.com/docs/guides/function-calling#function-tool-example

first client call with a tool definition "What is my horoscope? I am an Aquarius."
client receives the output, and runs the horoscope function
second client call with function tool output
verify the output looks good

Added the following as a test also.

Turn 1

curl http://localhost:20001/v1/responses   -H "Content-Type: application/json"   -N   -d '{
    "model": "/data/users/axia/checkpoints/gpt-oss-120b",
    "input": [],
    "previous_input_messages": [{
        "role": "system",
        "content": [
            {
                "type": "system_content",
                "model_identity": "You are ChatGPT, a large language model trained by OpenAI.",
                "reasoning_effort": "Medium",
                "conversation_start_date": "2025-10-06",
                "knowledge_cutoff": "2024-06",
                "channel_config": {
                    "valid_channels": [
                        "analysis",
                        "commentary",
                        "final"
                    ],
                    "channel_required": true
                }
            }
        ]
    },
    {
        "role": "developer",
        "content": [
            {
                "type": "developer_content",
                "tools": {
                    "functions": {
                        "name": "functions",
                        "tools": [
                            {
                                "name": "get_horoscope",
                                "description": "Get todays horoscope for an astrological sign.",
                                "parameters": {}
                            }
                        ]
                    }
                }
            }
        ]
    },
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "What is my horoscope? I am an Aquarius."
            }
        ]
    }],
    "temperature": 0.7,
    "max_output_tokens": 1000,
    "enable_response_messages": true, "stream": true
}'

Turn 1 output

{
    "response": {
        "id": "resp_d1bde89745cb45449c0377df8b3af921",
        "created_at": 1760654776,
        "incomplete_details": null,
        "instructions": null,
        "metadata": null,
        "model": "/data/users/axia/checkpoints/gpt-oss-120b",
        "object": "response",
        "output": [
            {
                "id": "rs_55e49b597c6640fc9a73b884701ecb09",
                "summary": [],
                "type": "reasoning",
                "content": [
                    {
                        "text": "User asks for horoscope for Aquarius. We can use function get_horoscope. Need to call it with sign Aquarius.",
                        "type": "reasoning_text"
                    }
                ],
                "encrypted_content": null,
                "status": null
            },
            {
                "arguments": "{\n  \"sign\": \"Aquarius\"\n}",
                "call_id": "call_d61059ae16db4c2f8c5a5f8b8e0cd6bd",
                "name": "get_horoscope",
                "type": "function_call",
                "id": "fc_d61059ae16db4c2f8c5a5f8b8e0cd6bd",
                "status": null
            }
        ],
        "parallel_tool_calls": true,
        "temperature": 0.7,
        "tool_choice": "auto",
        "tools": [],
        "top_p": 1.0,
        "background": false,
        "max_output_tokens": 1000,
        "max_tool_calls": null,
        "previous_response_id": null,
        "prompt": null,
        "reasoning": null,
        "service_tier": "auto",
        "status": "completed",
        "text": null,
        "top_logprobs": null,
        "truncation": "disabled",
        "usage": {
            "input_tokens": 130,
            "input_tokens_details": {
                "cached_tokens": 0,
                "input_tokens_per_turn": [
                    130
                ],
                "cached_tokens_per_turn": [
                    0
                ]
            },
            "output_tokens": 55,
            "output_tokens_details": {
                "reasoning_tokens": 30,
                "tool_output_tokens": 0,
                "output_tokens_per_turn": [
                    55
                ],
                "tool_output_tokens_per_turn": [
                    0
                ]
            },
            "total_tokens": 185
        },
        "user": null,
        "input_messages": [
            {
                "role": "system",
                "name": null,
                "content": [
                    {
                        "model_identity": "You are ChatGPT, a large language model trained by OpenAI.",
                        "reasoning_effort": "Medium",
                        "conversation_start_date": "2025-10-06",
                        "knowledge_cutoff": "2024-06",
                        "channel_config": {
                            "valid_channels": [
                                "analysis",
                                "commentary",
                                "final"
                            ],
                            "channel_required": true
                        },
                        "type": "system_content"
                    }
                ]
            },
            {
                "role": "developer",
                "name": null,
                "content": [
                    {
                        "tools": {
                            "functions": {
                                "name": "functions",
                                "tools": [
                                    {
                                        "name": "get_horoscope",
                                        "description": "Get todays horoscope for an astrological sign.",
                                        "parameters": {}
                                    }
                                ]
                            }
                        },
                        "type": "developer_content"
                    }
                ]
            },
            {
                "role": "user",
                "name": null,
                "content": [
                    {
                        "type": "text",
                        "text": "What is my horoscope? I am an Aquarius."
                    }
                ]
            }
        ],
        "output_messages": [
            {
                "role": "assistant",
                "name": null,
                "content": [
                    {
                        "type": "text",
                        "text": "User asks for horoscope for Aquarius. We can use function get_horoscope. Need to call it with sign Aquarius."
                    }
                ],
                "channel": "analysis"
            },
            {
                "role": "assistant",
                "name": null,
                "content": [
                    {
                        "type": "text",
                        "text": "{\n  \"sign\": \"Aquarius\"\n}"
                    }
                ],
                "channel": "commentary",
                "recipient": "functions.get_horoscope",
                "content_type": "<|constrain|>json"
            }
        ]
    },
    "sequence_number": 38,
    "type": "response.completed"
}

Turn 2

curl http://localhost:20001/v1/responses   -H "Content-Type: application/json"   -N   -d '{
    "model": "/data/users/axia/checkpoints/gpt-oss-120b",
    "input": [],
    "previous_input_messages": [{
        "role": "system",
        "content": [
            {
                "type": "system_content",
                "model_identity": "You are ChatGPT, a large language model trained by OpenAI.",
                "reasoning_effort": "Medium",
                "conversation_start_date": "2025-10-06",
                "knowledge_cutoff": "2024-06",
                "channel_config": {
                    "valid_channels": [
                        "analysis",
                        "commentary",
                        "final"
                    ],
                    "channel_required": true
                }
            }
        ]
    },
    {
        "role": "developer",
        "content": [
            {
                "type": "developer_content",
                "tools": {
                    "functions": {
                        "name": "functions",
                        "tools": [
                            {
                                "name": "get_horoscope",
                                "description": "Get todays horoscope for an astrological sign.",
                                "parameters": {}
                            }
                        ]
                    }
                }
            }
        ]
    },
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "What is my horoscope? I am an Aquarius."
            }
        ]
    },
    {
        "role": "assistant",
        "name": null,
        "content": [
            {
                "type": "text",
                "text": "User asks for horoscope for Aquarius. We can use function get_horoscope. Need to call it with sign Aquarius."
            }
        ],
        "channel": "analysis"
    },
    {
        "role": "assistant",
        "name": null,
        "content": [
            {
                "type": "text",
                "text": "{\n  \"sign\": \"Aquarius\"\n}"
            }
        ],
        "channel": "commentary",
        "recipient": "functions.get_horoscope",
        "content_type": "<|constrain|>json"
    },
    {
        "role": "tool",
        "name": "functions.get_horoscope",
        "content": [
            {
                "type": "text",
                "text": "{\"output\": {\"horoscope\": \"{\"sign\": \"Aquarius\"}: Next Tuesday you will befriend a baby otter.\"}}"
            }
        ],
        "content_type": "<|constrain|>json"
    }
    ],
    "temperature": 0.7,
    "max_output_tokens": 1000,
    "enable_response_messages": true, "stream": true
}'

"response": {
        "id": "resp_8d0b532ff74f443dbba13cfbda5fd1c8",
        "created_at": 1760655738,
        "incomplete_details": null,
        "instructions": null,
        "metadata": null,
        "model": "/data/users/axia/checkpoints/gpt-oss-120b",
        "object": "response",
        "output": [
            {
                "id": "rs_a21dd22daa67407586aafae9c065ffc3",
                "summary": [],
                "type": "reasoning",
                "content": [
                    {
                        "text": "The function returned a weird output: {\"horoscope\": \"{\"sign\": \"Aquarius\"}: Next Tuesday ...\"} This looks malformed. Likely the function returns an object with horoscope string. We need to parse it. The output field is a string that includes JSON-like inside. Probably the intended horoscope is \"Next Tuesday you will befriend a baby otter.\" Let's respond with that.",
                        "type": "reasoning_text"
                    }
                ],
                "encrypted_content": null,
                "status": null
            },
            {
                "id": "msg_0cf364c9e8a4439193937efa2b036449",
                "content": [
                    {
                        "annotations": [],
                        "text": "Here’s your Aquarius horoscope for today:\n\n**“Next Tuesday you will befriend a baby otter.”**",
                        "type": "output_text",
                        "logprobs": null
                    }
                ],
                "role": "assistant",
                "status": "completed",
                "type": "message"
            }
        ],
        "parallel_tool_calls": true,
        "temperature": 0.7,
        "tool_choice": "auto",
        "tools": [],
        "top_p": 1.0,
        "background": false,
        "max_output_tokens": 1000,
        "max_tool_calls": null,
        "previous_response_id": null,
        "prompt": null,
        "reasoning": null,
        "service_tier": "auto",
        "status": "completed",
        "text": null,
        "top_logprobs": null,
        "truncation": "disabled",
        "usage": {
            "input_tokens": 212,
            "input_tokens_details": {
                "cached_tokens": 176,
                "input_tokens_per_turn": [
                    212
                ],
                "cached_tokens_per_turn": [
                    176
                ]
            },
            "output_tokens": 111,
            "output_tokens_details": {
                "reasoning_tokens": 80,
                "tool_output_tokens": 0,
                "output_tokens_per_turn": [
                    111
                ],
                "tool_output_tokens_per_turn": [
                    0
                ]
            },
            "total_tokens": 323
        },
        "user": null,
        "input_messages": [
            {
                "role": "system",
                "name": null,
                "content": [
                    {
                        "model_identity": "You are ChatGPT, a large language model trained by OpenAI.",
                        "reasoning_effort": "Medium",
                        "conversation_start_date": "2025-10-06",
                        "knowledge_cutoff": "2024-06",
                        "channel_config": {
                            "valid_channels": [
                                "analysis",
                                "commentary",
                                "final"
                            ],
                            "channel_required": true
                        },
                        "type": "system_content"
                    }
                ]
            },
            {
                "role": "developer",
                "name": null,
                "content": [
                    {
                        "tools": {
                            "functions": {
                                "name": "functions",
                                "tools": [
                                    {
                                        "name": "get_horoscope",
                                        "description": "Get todays horoscope for an astrological sign.",
                                        "parameters": {}
                                    }
                                ]
                            }
                        },
                        "type": "developer_content"
                    }
                ]
            },
            {
                "role": "user",
                "name": null,
                "content": [
                    {
                        "type": "text",
                        "text": "What is my horoscope? I am an Aquarius."
                    }
                ]
            },
            {
                "role": "assistant",
                "name": null,
                "content": [
                    {
                        "type": "text",
                        "text": "User asks for horoscope for Aquarius. We can use function get_horoscope. Need to call it with sign Aquarius."
                    }
                ]
            },
            {
                "role": "assistant",
                "name": null,
                "content": [
                    {
                        "type": "text",
                        "text": "{\n  \"sign\": \"Aquarius\"\n}"
                    }
                ]
            },
            {
                "role": "tool",
                "name": "functions.functions.get_horoscope",
                "content": [
                    {
                        "type": "text",
                        "text": "{\"output\": {\"horoscope\": \"{\"sign\": \"Aquarius\"}: Next Tuesday you will befriend a baby otter.\"}}"
                    }
                ],
                "channel": "commentary"
            }
        ],
        "output_messages": [
            {
                "role": "assistant",
                "name": null,
                "content": [
                    {
                        "type": "text",
                        "text": "The function returned a weird output: {\"horoscope\": \"{\"sign\": \"Aquarius\"}: Next Tuesday ...\"} This looks malformed. Likely the function returns an object with horoscope string. We need to parse it. The output field is a string that includes JSON-like inside. Probably the intended horoscope is \"Next Tuesday you will befriend a baby otter.\" Let's respond with that."
                    }
                ],
                "channel": "analysis"
            },
            {
                "role": "assistant",
                "name": null,
                "content": [
                    {
                        "type": "text",
                        "text": "Here’s your Aquarius horoscope for today:\n\n**“Next Tuesday you will befriend a baby otter.”**"
                    }
                ],
                "channel": "final"
            }
        ]
    },

mergify · 2025-10-16T19:49:10Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @qandrew.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Andrew Xia <axia@fb.com>

qandrew · 2025-10-17T16:59:37Z

    get_system_message,
-    parse_chat_input,
    parse_chat_output,
+    parse_input_to_harmony_message,


no functional changes in this file, just renamed the function and formatted code

Signed-off-by: Andrew Xia <axia@fb.com>

qandrew · 2025-10-17T17:03:29Z

ready for review,
cc @chaunceyjiang (related to function tools) @yeqcharlotte @lacora @alecsolder

qandrew · 2025-10-17T17:13:54Z

/gemini review

gemini-code-assist

Code Review

This pull request adds support for previous_input_messages in ResponsesRequest to enable multi-turn conversations, which is a valuable addition. The implementation is mostly sound and is accompanied by new tests. However, I've identified a couple of high-severity issues. One is a bug in the new test case where the tool name is incorrect, potentially masking issues. The other is a significant code duplication in the message processing logic, which impacts maintainability. Addressing these points will improve the robustness and quality of the code.

Signed-off-by: Andrew Xia <axia@fb.com>

qandrew

@yeqcharlotte @alecsolder thanks for the comments! should be ready for another review

yeqcharlotte · 2025-10-24T05:55:50Z

+                    message_role = message.author.role
+                    # To match OpenAI, instructions, reasoning and tools are
+                    # always taken from the most recent Responses API request
+                    # not carried over from previous requests
+                    if (
+                        message_role == OpenAIHarmonyRole.SYSTEM
+                        or message_role == OpenAIHarmonyRole.DEVELOPER
+                    ):
+                        continue


move these logic to harmony util?

@yeqcharlotte thanks for the catch, done

Signed-off-by: Andrew Xia <axia@fb.com>

yeqcharlotte

thanks for addressing all the updates. it looks much cleaner now!

…ct#26962) Signed-off-by: Andrew Xia <axia@fb.com> Co-authored-by: Andrew Xia <axia@fb.com>

mergify Bot added frontend gpt-oss Related to GPT-OSS models labels Oct 16, 2025

github-project-automation Bot added this to gpt-oss Issues & Enhancements Oct 16, 2025

github-project-automation Bot moved this to To Triage in gpt-oss Issues & Enhancements Oct 16, 2025

qandrew force-pushed the input-messages-2 branch from 340b276 to 21815e5 Compare October 16, 2025 00:20

mergify Bot added the needs-rebase label Oct 16, 2025

initial commit input_messages

ec420f3

Signed-off-by: Andrew Xia <axia@fb.com>

qandrew force-pushed the input-messages-2 branch from 21815e5 to ec420f3 Compare October 16, 2025 20:51

mergify Bot removed the needs-rebase label Oct 16, 2025

function call round trip

82e94a4

Signed-off-by: Andrew Xia <axia@fb.com>

qandrew changed the title ~~Input messages 2~~ [gpt-oss][2/N] Support input_messages in responsesRequest Oct 16, 2025

Andrew Xia added 3 commits October 16, 2025 17:25

restructure so we use system/dev prompt from latest request

98c41db

Signed-off-by: Andrew Xia <axia@fb.com>

test harmony utils

2532ed0

Signed-off-by: Andrew Xia <axia@fb.com>

unit test

67a5045

Signed-off-by: Andrew Xia <axia@fb.com>

qandrew commented Oct 17, 2025

View reviewed changes

Andrew Xia added 2 commits October 17, 2025 10:01

clean

6995d10

Signed-off-by: Andrew Xia <axia@fb.com>

remove precommit

89061d4

Signed-off-by: Andrew Xia <axia@fb.com>

qandrew marked this pull request as ready for review October 17, 2025 17:02

qandrew requested review from DarkLight1337, NickLucche, aarnphm, chaunceyjiang, robertgshaw2-redhat and simon-mo as code owners October 17, 2025 17:02

qandrew mentioned this pull request Oct 17, 2025

[Frontend] Add OpenAI Harmony integration for responses API #24472

Closed

gemini-code-assist Bot reviewed Oct 17, 2025

View reviewed changes

Comment thread tests/entrypoints/openai/test_response_api_with_harmony.py

Comment thread vllm/entrypoints/openai/serving_responses.py Outdated

yeqcharlotte reviewed Oct 18, 2025

View reviewed changes

Comment thread vllm/entrypoints/harmony_utils.py Outdated

yeqcharlotte reviewed Oct 18, 2025

View reviewed changes

Comment thread vllm/entrypoints/openai/serving_responses.py Outdated

charlotte comments

ac078e2

Signed-off-by: Andrew Xia <axia@fb.com>

qandrew requested a review from yeqcharlotte October 20, 2025 16:29

yeqcharlotte reviewed Oct 21, 2025

View reviewed changes

Comment thread vllm/entrypoints/openai/serving_responses.py

yeqcharlotte reviewed Oct 21, 2025

View reviewed changes

Comment thread vllm/entrypoints/openai/protocol.py Outdated

yeqcharlotte reviewed Oct 21, 2025

View reviewed changes

Comment thread vllm/entrypoints/openai/serving_responses.py Outdated

alecsolder reviewed Oct 21, 2025

View reviewed changes

comments

b1d2b02

Signed-off-by: Andrew Xia <axia@fb.com>

qandrew force-pushed the input-messages-2 branch from 04fad7a to b1d2b02 Compare October 24, 2025 00:53

qandrew commented Oct 24, 2025

View reviewed changes

Comment thread vllm/entrypoints/openai/serving_responses.py

Comment thread vllm/entrypoints/openai/serving_responses.py Outdated

yeqcharlotte reviewed Oct 24, 2025

View reviewed changes

Andrew Xia added 2 commits October 24, 2025 19:47

move

cc0be1d

Signed-off-by: Andrew Xia <axia@fb.com>

Merge branch 'main' into input-messages-2

ce12450

qandrew requested a review from yeqcharlotte October 25, 2025 02:52

yeqcharlotte approved these changes Oct 27, 2025

View reviewed changes

github-project-automation Bot moved this from To Triage to Ready in gpt-oss Issues & Enhancements Oct 27, 2025

yeqcharlotte enabled auto-merge (squash) October 27, 2025 20:52

github-actions Bot added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 27, 2025

yeqcharlotte merged commit 53a56e6 into vllm-project:main Oct 27, 2025
49 checks passed

github-project-automation Bot moved this from Ready to Done in gpt-oss Issues & Enhancements Oct 27, 2025

ilmarkov pushed a commit to neuralmagic/vllm that referenced this pull request Nov 7, 2025

[gpt-oss][2/N] Support input_messages in responsesRequest (vllm-proje…

920348f

…ct#26962) Signed-off-by: Andrew Xia <axia@fb.com> Co-authored-by: Andrew Xia <axia@fb.com>

ZhengHongming888 pushed a commit to ZhengHongming888/vllm that referenced this pull request Nov 8, 2025

[gpt-oss][2/N] Support input_messages in responsesRequest (vllm-proje…

1abe0df

…ct#26962) Signed-off-by: Andrew Xia <axia@fb.com> Co-authored-by: Andrew Xia <axia@fb.com>

rtourgeman pushed a commit to rtourgeman/vllm that referenced this pull request Nov 10, 2025

[gpt-oss][2/N] Support input_messages in responsesRequest (vllm-proje…

7c819ab

…ct#26962) Signed-off-by: Andrew Xia <axia@fb.com> Co-authored-by: Andrew Xia <axia@fb.com>

devpatelio pushed a commit to SumanthRH/vllm that referenced this pull request Nov 29, 2025

[gpt-oss][2/N] Support input_messages in responsesRequest (vllm-proje…

d19e50f

…ct#26962) Signed-off-by: Andrew Xia <axia@fb.com> Co-authored-by: Andrew Xia <axia@fb.com>

Uh oh!

Conversation

qandrew commented Oct 16, 2025 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Uh oh!

mergify Bot commented Oct 16, 2025

Uh oh!

qandrew Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

qandrew commented Oct 17, 2025

Uh oh!

qandrew commented Oct 17, 2025

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

qandrew left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

yeqcharlotte Oct 24, 2025

Choose a reason for hiding this comment

Uh oh!

qandrew Oct 25, 2025

Choose a reason for hiding this comment

Uh oh!

yeqcharlotte left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

qandrew commented Oct 16, 2025 •

edited by github-actions Bot

Loading